Repository navigation
Write struct columns as Rust's vortex.struct layout - #739
Merged
Merged
Conversation
Each field of a struct column is its own chunked, repartitioned, dict-able column under a validity child when the struct is nullable; fields are zoned only while no struct above them is nullable, as in Rust. Fixes #724 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #724.
A struct column was one flat array (under
vortex.maskedwhen nullable). It is now Rust's struct layout: validity child (nullable only) then one child per field, each chunked/dict-encoded on its own. Fields are zoned only while no struct above is nullable (checked against vortex-jni).Tests:
StructLayoutInteropIntegrationTestcompares Java's layout tree with vortex-jni's (nullable and not) and has vortex-jni read Java's file; the two layout cases fail without the change../mvnw verifyis green.Not covered: a
ColumnEncodingoverride on a struct column is now ignored (fields are leaves); the mnist size gain is not measured.🤖 Generated with Claude Code